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Abstract 

There are theoretical models concerning Multimedia and cognitive and motivational 
processes. However, such models are missing for emotional processes. In order to 
develop such a model, studies from literary text research were analyzed. Literary text 
research was chosen as a starting point because Multimedia environments often 
contain texts (or stories) with an emotional quality. Analyzing and optimizing such 
texts or stories have a long tradition within this kind of research. The resulting model 
relates Multimedia elements, story elements (e.g., foregrounding), perceptual- 
cognitive-emotional processes (e.g., presence), and emotional outcomes (e.g., 
pleasantness) to each other. Theoretical implications concern the expansion of the 
model in respect to other Multimedia elements and to the integration with existing 
cognitive and motivational approaches. Methodological implications focus on 
suggestions for measurement development. Practical implications deal with 
Multimedia design and dramaturgical e-learning strategies. 

Multimedia, Stories, and Emotion - An Integrated Model for Research and 
Design 

Multimedia elements are fascinating people in computer games or web 
environments, not only because of their power to influence cognitive and motivational 
processes, but especially because of their capability to stimulate emotions, like fear 
or pleasure. How to arouse emotions with Multimedia represents an issue in research 
which is usually related to computer games. There is tremendous research on how 
computer games can produce emotions like, for example, aggression (e.g., Singer & 
Singer, 2002). However, this kind of research focuses mainly on negative emotions 
(e.g., anger) and on general long-term effects of different games or media settings 
(including TV). It shows little about which specific Multimedia elements produce 
positive influences on short-term emotional processes (e.g., Bryant & Zillmann, 
2002). To identify short-term effects on emotional processes is important because it 
can be assumed that long-term effects are built when continuously short-term effects 
are cumulated. In addition, it can be assumed that short-term effects can more easily 


be used for educational purposes (e.g., for establishing emotionally sound learning 
environments for children). 

Identifying short-term effects of Multimedia elements was done in the field of 
cognitive and motivational processes based on sophisticated theory building and 
related empirical research (e.g., Astleitner & Wiesner, 2004; Keller, 1999; Malone & 
Lepper, 1987; Mayer, 2001). However, such activities are lacking in respect to 
emotional processes. For example, recent comprehensive reviews of Multimedia 
effects neglected entirely the emotional perspective (e.g., Hede, 2002). Indeed 
Astleitner and Leutner (2000) presented a research-based approach dealing with the 
question of how computer-based environments can be made more emotionally 
sound. But, this approach is focusing on instructional elements on a macro-level 
(referring to emotional outcomes), and not dealing with Multimedia elements on a 
micro-level (including emotional processes). Also, the "emotioneering"-techniques 
from Freeman (2003) do not improve the situation, because they are not related to 
research, but to the practice of game design. Such techniques induce, for example, a 
computer player to identify with the role he or she is playing. The over 300 distinct 
techniques include, for example, ways to give emotional depth to a non-player 
character, even if the player has just one line of dialogue. 

Finding suitable basic research dealing with Multimedia and emotion can be based 
on a definition of Multimedia. According to Jonassen (2000, p. 207), the commonality 
among these definitions "involves the integration of more than one medium into some 
form of communication" and "refers to the integration of media such as text, sound, 
graphics, animation, video, imaging, and spatial modeling into a computer system". 
Referring to this definition, a first step of research can be to relate each element of 
Multimedia to emotional processes, and a second step, to combine some elements 
and analyze their joint effects on emotional processes achieving finally a 
comprehensive model of different Multimedia effects on emotion. There is, for 
example, research on sound (e.g., Bradley & Lang, 2000), on music (e.g., Juslin & 
Sloboda, 2001), and on video (e.g., Cennamo, 1993) and emotional processes. 

However, the way of establishing a first research-based link from Multimedia 
elements to emotional processes in this paper is to focus on text and especially on 
literary text research. There are several reasons for this decision: a) one of the most 
frequently stated reasons for reading literary text is to experience emotions or 
emotional processes, so there is a strong relationship between literary text and 
emotion (see, for example, fiction with high emotional appeal from Gass, 1999 or 
Obermayr, 1998); b) the main elements in many Multimedia environments (e.g., 
learning environments on CD-ROMS or on the Internet) are texts, because texts 
represent the major source of information and handling of texts do not require 
sophisticated technical equipment (Jonassen, 1999); c) there is a long tradition in 
research on literary text and emotion what increases the chance to identify a highly 
elaborated network of variables which can build a basis for Multimedia research and 
design (e.g., Oxenhandler, 1988); and d) there are reviews concerning media in 
general and emotions, however, they do not focus on literary text and/or Multimedia 
(e.g., Schwab, 2001). 

Based on literary text research, the following research questions should be answered 
within this study: 



a) Which elements of texts do trigger emotional processes? For analyzing 
literary texts, the following aspects are usually considered: theme, character, plot, 
structure, setting, point of view, language and style, etc. These aspects must be 
evaluated in their capacity to produce emotions in Multimedia based on empirical 
literary text research. 

b) Which emotional processes are triggered? According to Kleinginna and 
Kleinginna (1981, p. 355) emotions are "a complex set of interactions among 
subjective and objective factors, mediated by neural/hormonal systems, which can: 
(a) give rise to affective experiences such as feelings of arousal, 
pleasure/displeasure; (b) generate cognitive processes such as emotionally relevant 
perceptual effects, appraisals, labeling processes; (c) activate widespread 
physiological adjustments to the arousing conditions; and (d) lead to behavior that is 
often, but not always, expressive, goal directed, and adaptive". Emotional processes 
are based on the acceptance or rejection of objects and facts and have an 
evaluational relation to the world (Kuhl, 1986). Emotions, although they interact with 
cognitive and motivational processes, are considered in research as a unique 
component of human mental states, experiences, and behavioral expressions (e.g., 
Frijda, 1986). Emotions may initiate, terminate, or disrupt information processing. 
They may result in selective information processing or they may organize recall. 
"Feelings" are the "conscious recognition of emotions" (Greenberg & Snell, 1997, p. 
96). Emotional processes are interactive and numerous. It is the task of this paper to 
select these processes which turned out to be important in research and which are 
relevant for designing Multimedia environments. 

c) How can emotionally relevant elements of texts be used and implemented 
within Multimedia environments in order to improve their capacity to affect emotions? 
Such Multimedia environments allow for the electronically integrated display and user 
control of a variety of media formats and information types, including motion video 
and film, still photographs, text, graphics, animation, sound, numbers, and data. The 
resulting experience for the user is a multidimensional, multi-sensory interweave of 
self-directed reading, viewing, listening, and interacting, through activities such as 
exploring, searching, manipulating, writing, linking, creating, juxtaposing, or editing 
(e.g., Wilson 1992). Within this paper, however, only texts and their emotional 
capacities are analyzed. For example, from text comprehension research it is well 
known that learning from text and learning from, for example, pictures are 
qualitatively different ways of constructing mental models. While text comprehension 
is an indirect transformation between a text as a symbolic representation and the 
mental model as an analog representation, picture comprehension is a direct 
mapping of the picture as an analog representation onto the mental model as an 
analog representation by establishing an analogy between the visual information and 
the corresponding mental model (e.g., Chun & Plass, 1997). There exists no such 
comparable research for emotional processes. That is the reason why exclusively 
texts are analyzed in a first step of research. 

All the questions should be answered on the basis of a narrative literature review 
concerning results of empirical studies. This literature review is based on some 
general assumptions about emotions and media (e.g., Mangold, Unz, & Winterhoff- 
Spurk, 2001). These assumptions are: Confrontations with media contents produce 
different intensities of emotional experiences. These emotions refer to situations 
situated within the media contents but also to real life situations which are, to some 
extent, similar to the media contents. Emotional experiences can be related to one, 
but also to different multiple emotions. Emotions can change rapidly in time. Finally, 



different people experience different emotions if confronted with the same media 
contents. 


According to a taxonomy of literature reviews from Cooper and Hedges (1994), the 
focus of the review concerning literary text research and emotion was on practices or 
applications in the field of reading, on the identification and integration of central 
issues, on a representative (but not exhaustive) coverage of literature, on a 
conceptual organization of research results, and on Multimedia researchers in the 
field of education and psychology as target group. 

Despite this anchoring in existing empirical research, it has to be stressed that the 
resulting review do not represent definite generalizations of empirically well-founded 
research results for Multimedia designers, but nothing more than hypothetical 
prescriptions which should guide researchers and practitioners in their daily work 
concerning research on and design of emotionally sound Multimedia. Although many 
of these hypothetical prescriptions refer to results found in empirical research, they 
also have a speculative element delivering an inspiring force for future Multimedia 
research and design. For that purpose, it is not useful to distinguish between 
empirically well-founded and not well-founded results. All results, even the ones 
related with some empirical evidence, should be seen as hypotheses, because they 
are part of a systematic and mingled framework which has not been tested yet in the 
field of Multimedia. 

A Model of Multimedia, Story Elements, and Emotions 

Figure 1 shows the variables which relates Multimedia elements to emotional output 
variables based on literary text research results. The identification of these variables 
was based on research from Alfes (1995), Appel, Koch, Schreier, and Groeben 
(2002), De Vega, Leon, and Diaz (1996), Lombard and Ditton (1997), Oatley (1994), 
and Miall and Kuiken (1994, 2002). Relations between these variables are 
hypothesized related to research from Dijkstra, Zwaan, Graesser, and Magliano 
(1994), Gygax, Oakhill, and Garnham (2003), Kneepkens and Zwaan (1994), 
Levorato (2003), Sadoski, Goetz, and Rodriguez (2000), and Van Reekum (2000). 
These research results were used to build a model of Multimedia, story elements, 
and emotions. Within a first step, important components from research were 
selected. Within a next step, their position within the relationship between Multimedia 
elements and emotional outcomes were fixed. Then, research results were used to 
connect components as hypotheses. Finally, the model was evaluated (e.g., by 
proofing for inconsistencies) and calibrated (e.g., by improving definitions of 
concepts). 

Components of the Model 

Visual and auditory inputs build Multimedia elements which can be related to certain 
text or story elements. Such text or story elements which can be part of a Multimedia 
environment produce different cognitive-emotional processes which finally result in 
emotional outcomes. 

Figure 1 . Multimedia, story elements, and emotional processes. 
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Multimedia elements 

Visual input can take the form of text, pictures, diagrams, video, or animation. 
Auditory input can consist of sound, signals/cues, music, narration, or instructions. 
Multimedia elements are used in combination within Multimedia environments to 
produce "stories". 

Story elements 

Literary text elements build "stories", i.e., pieces of information that contain a chain of 
related fictional and/or non-fictional events. They can entertain, inform, or instruct 
people. Texts for entertainment are, for example, novels. Texts for information are, 
for example, reports in newspapers about political issues, or documentations in TV. 
Texts for instruction are, for example, contained in chapters of text books or teaching 
software on CD-ROM. 

On a macro-level, literary texts show - as a rule - a certain dramaturgy consisting of 
exposition (i.e., an explanation of something), confrontation (i.e., a situation in which 
a problem appears and needs to be dealt with), and solution (i.e., the correct answer 
to a problem). Typical literary text elements on a micro-level are: a) theme (idea or 
point of a story formulated as a generalization); b) character (imaginary or real 
people who act within the story); c) plot (arrangement of ideas and/or incidents that 
make up a story); d) structure (design or form of completed actions); e) setting (place 
or location of an action as well as the historical and cultural context for characters); f) 
point of view (pertains to who tells the story and how it is told); and g) foregrounding 
(stylistic effects at phonetic, grammatical, or semantic level; e.g., irony (a contrast or 
discrepancy between one thing and another), imagery (a concrete representation of a 
sense impression, a feeling, or an idea which appeals to one or more of our senses), 
symbolism (when objects or actions means more than themselves), allegory (a form 
of narrative in which people, places, syntax, diction (denotations and connotations), 
and events seem to have hidden meanings), and voice/sound (e.g., rhythm and 
meter)) (e.g., Jordan-Henley, 1999). 




Cognitive-emotional processes 

It is assumed that Multimedia and story elements are processed within working 
memory based on three types of processes: perceptional-cognitive, cognitive- 
emotional, and emotional. 

(1) First, Multimedia and story elements are evaluated within a perceptional- 
cognitive process. The evaluation process is based on different aspects: Sensory 
arousal concerns the number of human senses which are stimulated by the 
Multimedia and story elements. Realism refers to the degree of correspondence with 
reality. Dynamics concern the level of changes within an environment. Coherence is 
achieved when sequences of the text have something in common like similarities in, 
for example, time or place, or if they have a causal relationship. Novelty is the quality 
of being new. Relevance means the importance of a stimulus or event to one's goals 
and concerns. Valence is the pleasantness or unpleasantness of the stimulus or 
event. Difficulty concerns the estimated own ability to deal with the event and its 
consequences. Legitimacy represents the evaluation of one's own actions in relation 
to moral standards, social norms, or one's self-ideal. 

(2) Then, a cognitive-emotional process takes place in which cognitive and 
emotional conditions of feelings are controlled and processed. Presence concerns a 
feeling of being part of an environment. Emotional tracking means the representation 
of changing emotional states of characters. Emotional motivation is dealing with the 
control for achievement of emotional goals (i.e., entertainment, relaxation, 
stimulation, suspense). Sympathy refers to feelings for a character, empathy to 
feelings with a character. 

(3) Finally, emotional processes result in general feelings towards different 
aspects of the Multimedia and story elements. Aesthetic feelings are based on formal 
components of Multimedia or story elements (e.g., foregrounding). Narrative feelings 
concern feelings towards the sequence of events (e.g., resulting in increasing the 
empathy with a character). Evaluative feelings refer to feelings towards the text as a 
whole (e.g., overall enjoyment). Self-modifying feelings represent feelings that 
restructure own's understanding and sense of the self. 

Emotional outcomes 

They concern emotions, especially their pleasantness or unpleasantness (positive or 
negative value) and their activation or deactivation (energetic or lethargic arousal). In 
this way, emotions are characterized in terms of continuous dimensions, rather than 
as discrete categories. Emotions or emotional states can be easily placed in an 
emotional space defined by these dimensions (e.g., Vastfjall & Kleiner, 2002, p. 6), 
for example, "surprised" (activation), "excited" (pleasant activation), "happy" 
(pleasantness), "content" (pleasant deactivation), "passive" (deactivation), "bored" 
(unpleasant deactivation), "sad" (unpleasantness), or "annoyed" (unpleasant 
activation). 

Relationships between Components of the Model 

In general, there are multiple relationships between variables which are depicted in 
Figure 1. At the moment, it is not possible to relate systematically, comprehensively, 
and exclusively different independent variables (i.e., Multimedia and story elements) 
to mediating (i.e., perceptional-cognitive-emotional processes) and to dependent 
variables (i.e., activation and pleasantness). 

However, there are some general relationships combining groups of variables. It is 
assumed that Multimedia elements can be used to design certain story elements. 



These elements influence perceptional, cognitive, and emotional processes. It can 
also be hypothesized that perceptional processes influence cognitive processes and 
that cognitive processes affect emotional processes. Emotional processes should be 
accumulated and result in emotional outcomes. Finally, it is also highly probable that 
emotional outcomes also influence perceptional, cognitive, and emotional processes. 

In particular, the following specific relationships between the variables are assumed 
based on results of empirical research. Levorato (2003) found that there was a 
relationship between imagery and the perception of novelty, and also between 
novelty and emotional outcomes (e.g., pleasure). Alfes (1995) suggests on results of 
a qualitative study that the difficulty of a text in respect to understanding produces 
evaluative feelings and negative emotional outcomes (e.g., anger). The author also 
identified self-modifying feelings as control mechanism during reading. Others found 
that difficulty (concreteness) and relevance (familiarity) were related to a pleasant 
and active emotional outcome (interestingness) (Sadoski, Goetz, & Rodriguez, 
2000). Results from Gygax, Oakhill, and Garnham (2003) suggest that the emotional 
information readers infer from stories is too broad to determine specific emotions. 
The results are consistent with the idea that a general emotional response is evoked 
which is compatible with one or more specific emotions. These results support the 
assumption (of this paper) that especially emotional outcomes can be described 
along few dimensions. A review from Lombard and Ditton (1997) shows that 
presence is associated with sensual arousal, realism, and pleasant activation (i.e. , 
enjoyment). De Vega, Leon, and Diaz (1996) found that the salience and accessibility 
of emotional representations depend on the amount of contextual cues provided 
throughout a text. Once an emotional representation was built, guided by the initial 
plot of a story, the incoming information is checked against it. Results from a study by 
Dijkstra, Zwaan, Graesser, and Magliano (1994) suggest that sympathy occurs most 
strongly in text segments to the extent that these conveyed suspense (dynamics), 
described plausible events (realism), were important (relevance), contained dialogue, 
and occurred towards the end of the story (foregrounding, structure). In a study by 
Miall and Kuiken (1994), it is pointed out, that text segments with more foreground 
features (on a phonetic, grammatical, and thematic level) took longer to read, were 
found more striking, and evoked more affect. In explaining these results, the authors 
assumed that foregrounding prompts defamiliarization, (which is related to difficulty 
and coherence), defamiliarization evokes affect, and affect guides refamiliarizing 
integrative efforts. Also, Miall and Kuiken (2001) stress, based on their results, that a 
reader’s perspective is repeatedly disrupted and reshaped, especially in response to 
foregrounding (e.g., using metaphors). 

All these exemplary results can be anchored with the presented model as they 
concern relationships between different variables. They deliver - to some degree - a 
first validation of the model. 

implementation Strategies: How to Use the Model in Multimedia Research and 
Design 

The model can be used to stimulate research and practice in different ways: 

a) Theoretical implications: It builds a first network of hypotheses which can 
guide the development of existing and future theoretical models on Multimedia 
environments. The model can be expanded by other Multimedia elements (like 
pictures, sound, or video) and therefore build a comprehensive basis for testing 



emotional effects of Multimedia environments. The model can also be expanded in a 
way as it can be integrated within existing cognitive and motivational models of 
Multimedia effects (e.g., Astleitner& Wiesner, 2004; Mayer, 2001). 

b) Methodological implications: The model shows a list of independent and 
dependent variables which represent possible emotional effects of Multimedia 
environments. Such a list can be used for instrument development resulting, for 
example, in a questionnaire measuring emotional processes and outcomes when 
dealing with a Multimedia environment. 

c) Practical implications: The model builds a framework for solving emotional 
problems in Multimedia environments. The model can tell Multimedia designers 
which emotional processes can be influenced by which (story-/text-related) elements 
of a Multimedia environment. 

The variables which are depicted in Figure 1 can be seen as a basis for designing 
"emotionally sound" Multimedia (e.g., Lee & Boling, 1999). "Emotionally sound" 
means that positive emotions are increased and that negative emotions are 
decreased during handling a Multimedia environment. When using this basis, the 
Multimedia designer has to analyze emotional goals (i.e. , increasing or decreasing 
activation and/or pleasantness) and emotional problems which are experienced from 
users before and during performing within a Multimedia environment. Based on the 
observed problems, different Multimedia elements have to be designed and changed 
in or implemented into a Multimedia environment. Then the consequences of the 
implementation have to be evaluated. 

For example, the variables which were identified within this paper can be used and 
integrated within the "Dramaturgical E-learning Strategy (DES)" by Modinger and 
Thissen (2004). Within DES, tasks are transformed into conscious and unconscious 
spheres of experience using a dramaturgical basic structure (i.e., exposition, 
confrontation, and solution), cryptic knowledge, and community (i.e., a social- 
communicative context). Problem solving tasks which are embedded within a 
dramaturgical basic structure are assumed to be solved with knowledge modules 
which are available in form of hypertexts which include texts, but also other 
Multimedia elements. 

Discussions 

The model of literary text elements, Multimedia, and emotion focuses on literary text 
research in order to identify variables which influence emotions when handling a 
Multimedia environment. Relevant research from emotion and non-fictional texts, 
film, video, music, sound, intelligent agents, computer-mediated communication, 
web-based training, etc. was not considered (e.g., Astleitner, 2000; Astleitner, 2001; 
Lepper & Chabay, 1987; Schiefele & Krapp, 1996; Tan, 1996). Also, research 
concerning the "flow"-experience and literary texts was not included within this study, 
because this kind of research is significantly situated in the field of motivational 
research (e.g., Appel, Koch, Schreier, & Groeben, 2002). 

Important questions are unanswered so far, and represent major starting points for 
future activities in research and design: 

a) Do Multimedia elements used in cognitive and motivational Multimedia 
design also improve emotional problems? If this is the case, the presented elements 
and processes are more or less superfluous. In this respect, it has also to be proved 
whether the proposed story and Multimedia elements really influence the related 
emotional processes. 



b) Are all variables proposed in this article necessary to establish emotionally 
effective Multimedia? There may be ones which are more effective than others or 
which cover the effects of others. The proposed story and Multimedia elements have 
to be compared in their effects. 

c) How changes emotional processes if different story and Multimedia 
elements are used in combination? Emotional processes are complex and 
dynamically interacting with each other. From this characteristic, it seems to be 
necessary that Multimedia elements must be combined in order to produce 
significant emotional effects. But, when combining different elements, unknown 
effects might occur that have to be given special attention. There also might be an 
overlapping with elements from cognitive and motivational design of Multimedia 
which can produce unintended side effects. Such disturbances resulting from 
implementing at one time different types of story and Multimedia elements have to be 
clarified. 

d) Which story and Multimedia elements are effective and efficient in 
influencing emotional processes? Of course the proposed elements have to be 
tested whether they are effective, but they should also be evaluated concerning their 
efficiency. For example, the realization of some elements in Multimedia design might 
be time consuming and can intensify information overload what has to be avoided. 

e) Which story and Multimedia elements should be applied in which situation 
and for which type of persons? Story and Multimedia elements do not work for all 
people in all kinds of situations. There has to be research for defining the situational 
and personal conditions for the application of particular elements in a way of 
"emotionally adaptive Multimedia environments" (e.g., Song & Keller, 2001). 

Despite these open questions, the presented article should motivate Multimedia 
designers and researchers to consider emotional processes more systematically and 
on a larger scale. 
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